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ABSTRACT 


Budget limitations have redueed the number of military components available for 
testing, and time constraints have reduced the amount of time available for actual testing 
resulting in many items still operating at the end of test cycles. These two factors 
produce small test populations (small sample size) with heavily censored data. The 
assumption of normality for estimates based on these small sample sizes reduces the 
accuracy of confidence bounds of the probability plots and the associated quantities. This 
creates a problem in acquisition analysis because the confidence in the probability 
estimates influences the number of spare parts required to support a mission or 
deployment or determines the length of warranty ensuring proper operation of systems. 
This thesis develops a method that simulates small samples with censored data and 
examines the error of the Fisher-Matrix (FM) and the Likelihood Ratio Bounds (LRB) 
confidence methods of two test populations (size 10 and 20) with three, five, seven and 
nine observed failures for the Weibull distribution. This thesis includes a Monte Carlo 
simulation code written in S-Plus that can be modified by the user to meet their particular 
needs for any sampling and censoring scheme. To illustrate the approach, the thesis 
includes a catalog of corrected confidence bounds for the Weibull distribution, which can 
be used by acquisition analysts to adjust their confidence bounds and obtain a more 
accurate representation for warranty and reliability work. 
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EXECUTIVE SUMMARY 


Shrinking budgets have affected all aspects of military hardware testing including 
the number of items that are procured for test in order to establish and verify their 
reliability. The amount of time given to test specific items has been reduced to shorten 
the acquisition time. These two factors contribute to fewer numbers of test items put into 
a test scenario with even a smaller number of failures occurring during the allotted test 
time. 

The few failures that occur during the test period provide valuable information to 
analysts, but the ones that don’t fail (the survivors at the end of the allotted test time) may 
provide even more information. The items that survive the test period are referred to as 
“censored” and in particular, the items whose testing begins at the same time and ends 
when the allotted test time is completed is referred to as “time-censored” or Type I 
censored data. Another common type of censored data is “failure censored” or Type II 
and may be thought of as ending the test when a predetermined number of failures have 
occurred. Estimates based on Type II censoring are often used to generate estimates 
under Type I censoring because the tester can control the number of failures and hence, 
the number of censored items. 

Many current software packages such as Weibull++, S-Plus, and Minitab allow 
the analyst the ability to specify the total number of test items put into test, the times of 
the failures for any items, and the number of censored items (items still running at the 
end of the test time). The software packages listed above are based on large sample 
theory, yet users readily apply the analysis to a small sample size with heavily censored 
data with little regard to the inherent error. 

This assumption of normality due to large sample size, when applied to testing 
based on small sample sizes, has been known for decades to provide errors in the system 
reliability estimates and the confidence associated with those estimates. These errors 
were tolerated as minor, but in today’s economy of doing more with less, the government 
must be a good steward of its money and demand more accurate results from the limited 
assets it’s given. 
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The confidence associated with a system reliability estimate influences the number 
of spare parts required to support a mission or deployment or determines the length of 
warranty ensuring proper operation of systems. To ensure accurate confidence in small 
sample testing, this thesis develops a Monte Carlo methodology for the Weibull 
distribution, simulating small samples with censored data and examines the error of two 
commonly used methods to determine confidence bounds for reliability: 1. the Fisher- 
Matrix (FM) method and 2. the Likelihood Ratio Bound (LRB) method. Tables are 
provided that summarize the graphical output of the simulation code and quantify the 
difference between the two methods, with the end result of providing the user a more 
accurate method. 

A simulation method is used in this thesis to quantify the bias in confidence bounds 
for reliability that one obtains when using software packages based on large sample 
theory and applies that analysis to small sample size tests. A simulation of two small- 
sample test populations (sample size of 10 and 20) is used with three, five, seven and nine 
observed failures for the Weibull distribution. Type II censoring is used during the 
simulation by specifying the number of failures that occur out of the total sample size. 
The S-Plus code by default performs a 10000 iteration, Monte Carlo simulation for each 
test case and produces a graph for the more often used Fisher-Matrix (FM) confidence 
method. A catalog (Appendix B) is developed that can provide the user a more accurate 
confidence that they can use in their acquisition documents for many common 
reliabilities and censored data combinations. 

A detailed process is presented that leads a user through an example clarifying the 
simulation process. This example provides an avenue for the user to take advantage of 
the attached catalog showing several common reliability levels and confidence bounds 
for the Weibull distribution, and allows them to literally ‘pick-off the adjusted 
confidence bounds for their particular situation. In addition, the documented S-Plus code 
is included that can be modified to meet the users requirements allowing the graphs to be 
tailored to specific needs. 

The errors produced by the naive use of large sample methods are substantial when 
applied to small samples. A requested 90% lower confidence bound on reliability in 
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typical software packages may only provide an aetual 75% eoverage for small samples. 
This thesis eorreets that error by allowing the user to request a eorrected nominal 
eonfidenee level for small sample testing to achieve the true desired eoverage when using 
software paekages that are based on the “large sample theory”. For example, we may 
need to request a 99.2% eonfidenee level to aetually get a 90% eoverage, for a given 
sample size and eensoring seheme. 

This ability to eorreet for the error in the eonfidenee levels allows us to eontinue to 
use existing software tools, but to use them better. 


xvii 
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1. INTRODUCTION 


A. OVERVIEW 

Confidence intervals (Cl) in life data analysis are an indication of the degree of 
confidence that the analyst presents along with the statistical data. It is his “confidence” that a 
specific interval contains the quantity he is interested in. As stated quite nicely in their text on 
reliability methods, Meeker and Escobar said [Ref: 3, p.49], 

A specific interval either contains the quantity of interest or not; the truth is 
unknown. 

Therefore, the confidence interval quantifies the uncertainty, but as Nelson states in his 
book on applied life data analysis [Ref: 4, p.l96]. 

The real uncertainty is usually greater than the confidence interval indicates 
because the interval is based on certain assumptions about the data and the 
model.. .Departures from the assumptions add to the uncertainties in the results. 

Most statistical analysis (and life data analysis) assumes asymptotic normality of 
estimators based on large sample size theory, but as seen in recent acquisition testing, the 
government rarely has large numbers of test articles. Small sample sizes are especially prevalent 
in Developmental Testing (DT) and Operational Testing (OT) in procuring, testing, and buying 
major weapon systems. In addition to a small number of test articles that are provided for 
testing, the contractors generally have good in-house reliability programs that result in few 
failures during the test period; this leads to censored data. 

Even if a sufficient number of test items are available to meet the criteria of the large- 
sample size theory, the number of failures need to be large [Ref: 3, p.l86], not just the number of 
test items. As Nelson points out in his book on accelerated testing [Ref: 1, p.236]. 

Such asymptotic theory is also called large-sample theory. This terminology is 
misleading for multiply and singly censored data, since the number of failures 
needs to be large. In practice, the asymptotic theory is applied to small samples, 
since crude theory is better than no theory. 

This error in confidence bounds for system reliability using small sample sizes has been 
known and accepted for years. The impact of these errors in today’s climate of ‘doing more with 
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less’ and ‘the sooner the better’ impacts all aspects of life cycle cost estimation and possibly 
even mission capability. Budget limitations and time constraints force the tester to use a smaller 
number of test items with less time to test them, which in turn results in a small sample size that 
is heavily censored. The analyst must then apply statistical methods that are inherently biased 
for small samples. In his text on probability and statistics, Devore reflects on his concerns about 
using asymptotic methods on small samples when the distribution is non-normal [Ref; 5, p.302]. 

It would certainly be distressing to believe that your confidence level is about 

95% when in fact it is really more like 88%! 

The government as a consumer should get the confidence it pays for and this thesis 
allows for the correct adjustment to the contractual documents before products are purchased. 
Most commercial software packages (Weibull ++, Minitab, WinSmith, S-Plus) available for life 
data analysis use a normal-approximation method to determine the confidence intervals even for 
small sample sizes because the large-sample approximations are computationally easy. [Ref: 3, 
p. 186, Ref: 6, p. 135]. 

Many researchers have documented that a bias exists with small samples and have 
focused on finding new methods to improve the coverage of confidence intervals for reliability 
estimates(e.g. Nelson (1982), Piergorsch (1987), Ostrouchov and Meeker (1988), Vander Weil 
and Meeker (1990), Doganaksoy and Schmee (1993), Shao and Tu (1995), Jeng and Meeker 
(2000) [Ref: 6, p.135-136]). 

This thesis suggests using existing methods better by correcting for the bias in existing 
commercial software products. 

B. PURPOSE 

The purpose of this thesis is to document the error in confidence interval coverage for 
reliability estimates based on data from a Weibull distribution between the Fisher-Matrix (FM) 
and Likelihood Ratio Bounds (LRB) methods by conducting a simulation study and constructing 
graphs and tables that indicate the extent of the problem. A 10000 iteration, Monte Carlo 
simulation will be used to produce graphs that analysts can use to convert the nominal (1-a) 
100% 1-sided confidence bounds into confidence bounds with actual (1-a) 100% coverage. In 
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addition, the S-Plus code using the embedded SPLIDA routine [Ref: 2] that was written for the 
simulation study is included in Appendix A and allows users to tailor a simulation to fit their 
specific needs. 

Once the error is known, it can be corrected by applying an appropriate adjustment, 
resulting in the desired coverage level. This correction is determined both graphically and 
numerically. 

C. SCOPE AND LIMITATIONS 

Most texts on life data analysis related to reliability and warranty calculations suggest 
that using the large sample approximations to get approximate confidence intervals is good for 
quick and easy calculations [Ref: 3, p.l65], but more accurate procedures are required for formal 
proposals and final reports. This thesis performs a Monte Carlo simulation study using 10000 
iterations determining actual ‘coverage probabilities’ of confidence intervals for system 
reliability based on small samples and heavily censored data. These actual coverage 
probabilities can be used to correct the error in the asymptotic methods, providing the accurate 
procedures desired for formal proposals and final reports. 

This simulation method works for any location-scale or log-location-scale distribution. 
This thesis uses Weibull for simplicity, but the simulation code can be adapted for lognormal, 
normal, smallest extreme value, exponential, and logistic distributions. 

The thesis focuses on using a simulation code to quantify the error between the two 
confidence methodologies that were developed for large samples but are applied to small 
samples and will be limited to calculating the exact lower, one-sided confidence interval for the 
following test cases: 

Typical reliability values of 99, 98, 97, 96, 95, and 90 will be used in the simulation 

study. 

Small sample sizes of 10 and 20 with 3, 5, 7, and 9 observed failures will be used for 
each reliability value. 

The Weibull location-scale distribution with fixed parameters will be used exclusively for 
this simulation study. 
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In addition, the Fisher-Matrix (FM) bounds and Likelihood Ratio Bounds (LRB) methods 
will be eompared to obtain the bias in eonfidenee interval eoverage. Results of the eomparison 
will be doeumented in tables and will establish whieh method has the most error. A eatalog of 
the method with the most error will be attaehed as Appendix B for user eonvenienee. 

And finally, the S-Plus eode for the Monte Carlo simulation will be attaehed as Appendix 
A allowing users to tailor the simulation output for their speeifie applieation. The attaehed S- 
Plus eode also provides the user a subroutine that states the exaet eonfidenee level they must 
request requested in order to obtain the desired eoverage, for a given sampling size, eensoring 
seheme, and distribution. This eliminates the need to estimate the required value using the 
graphs in Appendix B. 
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II. BACKGROUND 


A. LIFE DATA ANALYSIS 

Life data analysis seeks to understand the data at hand obtained from a sample of test 
articles (descriptive statistics) and then apply the results to the entire population of test articles 
(inferential statistics). This section is provided as background material to focus the reader on the 
specific terms, procedures and definitions used in this thesis and provide a common background 
for users who want to tailor the S-Plus code for their specific application. 

I. Confidence Bounds Defined 

To compute a confidence interval from a given data set, you must first specify a 
confidence level. For example, if a data set consists of n observations assumed to have a normal 
distribution with known variance and unknown mean and if you want to find the 95% confidence 
interval for the mean, the equation for the confidence interval endpoints (confidence bounds) 
would look like (lower endpoint, upper endpoint) and is represented mathematically as: 


/_ 

X - 
V 


^a/2 




cr - 


—l=,X + z 


a/2 



( 2 . 1 ) 


where a is the known standard deviation, x is the mean of the data, a is (1 - .95) and n is the 
number of observations. The value of 1.96 (used for is obtained from the standard normal 

probability tables and represents the critical value under the normal curve that leaves the center 
area equal to .95 (the level of confidence desired). The area under the tails is the (1 - .95)* 100% 
= 5% that is equally divided for symmetric distributions (shown in Figure 1). The equal 
distribution of the confidence on either end of the probability curve displayed in Figure 1 is only 
correct for a symmetric distribution such as the normal family. 

According to Devore in his text on probability and statistics [Ref: 5, p. 281], the correct 
interpretation of this 95% confidence bound is true only over the long run. 

To say that an event A has probability (of) .95 is to say that if the experiment on 
which A is defined is performed over and over again, in the long run A will occur 
95% of the time. 
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In other words, the 95% eonfidenee level is not so mueh a statement of faet about any 
speeifie interval, but represents what would happen if a large number of similar events took 
plaee. This statement leads to the Monte Carlo technique of using 10000 iterations in the 
simulation study used in this thesis. 



Figure 1. Two-sided 95% confidence interval symmetrically distributed around the mean 

for a normal distribution. 

2. One-Sided Confidence Bounds 

Most reliability and warranty analysts use a one-sided confidence interval for reliability 
[Ref: 5, p.292] because they are interested in estimating the reliability with some confidence that 
the actual value is at least as large as that lower bound. In terms of risk, a one-sided interval 
establishes, with some confidence, the largest (or smallest) plausible value. For a symmetric 
distribution like the normal curve, the adjustment between a two-sided and a one-sided Cl is 
made by replacing with and choosing the appropriate end of the distribution (either the + 

or the - in Eq. 2.1). Now the single area to the right of the critical value 1.64 represents (1 - 
.95)* 100% = 5% as shown in Figure 2. 

3. Non-Symmetric Life Distributions in Small Samples 

When the sample size is small, the sampling distribution of an estimator may be skewed 
(a non-symmetric distribution) [Ref: 3, p.56] and using the normal approximation for confidence 
levels near either tail of the distribution may give a substantial error. 
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Figure 2. One-sided upper 95% confidence interval for the mean for a normal distribution. 

4. Censoring 

When several items are put into a test environment, either simultaneously or sequentially, 
not all of the units may fail by the end of the test. The survivors (unfailed units) are called 
censored, in that incomplete failure information is available; you only know they were still 
functioning at the end of the allotted test time. Type I censoring is generally considered as 
removing unfailed units from a test at a prescribed time (i.e. at the end of the allotted test time). 
Type I is also called “time censoring” by some authors and is considered to be the more common 
type of censoring in reliability data analysis. [Ref: 3, p.7; Ref: 6, p.l] Type II censoring is called 
“failure censoring” and is based on ending the test when a predetermined number of failures 
occur. 

5. The Weibull Distribution 

The Weibull distribution is a parametric distribution that is valuable in modeling certain 
life data. The benefit of using parametric models is that the distribution can be expressed 
entirely by just a few parameters. Caution should be used in referencing texts in probability and 
statistic theory that explain the Weibull distribution as the notation representing the shape and 
scale parameters are not standardized. This thesis uses the parameters P and rj for the shape and 
scale respectively as is shown in Meeker and Escobar’s text on statistical methods for reliability 
data [Ref: 3, p. 85]. The following equation is the probability density function (pdf) for the 
Weibull distribution using P and rj\ 


= ' exp 

rjP 




p~ 


t 


1 

.n) 

\ 


, t > 0, and /3,ri>0 


( 2 . 2 ) 
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Integrating the pdf with respeet to the independent variable (f) yields the easily reeognized 
eumulative distribution funetion (edf) for the Weibull distribution: 


F(t) = \- exp 




p~ 


t 


1 


\ 


, t>0 


(2.3) 


The system reliability R(t) is the probability that an item survives past time t and is defined as 
R(t) = l-F(t). 



Figure 3. Weibull probability density funetions for different shape parameters with a fixed 

scale parameter. 

Figure 3 shows that changing the value of the shape parameter allows the Weibull 
distribution to model many different types of behavior. The Weibull distribution can have either 
a decreasing or an increasing hazard function depending on the choice of parameters making it 
the “workhorse” of life data analysis because of its wide application. [Ref: 3, p.l73] 
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The hazard function is defined as the “propensity for an item to fail in the next small 
interval of time, given that the item has survived until some time (t)” [Ref; 3, p. 28] and is given 
by the equation h(t) =f(t)/(l-F(t)), where f(t) is the probability density function (pdf) and F(t) is 
the cumulative distribution function (cdf). The following set of plots in Figure 4 use the same 
shape parameter values as the Weibull probability density functions above, but displays the 
hazard function. 



Figure 4. Weibull hazard functions showing the effect of changing the shape parameter 

with a fixed scale parameter. 

6. Fisher-Matrix Method 

Confidence intervals (and hence the confidence bounds) can be calculated for parameters 
of the Weibull distribution in a manner similar to that shown for the normal distribution in 
Equation 2.1. Typical software packages for life data analysis use the normal-approximation 
method because it’s fast and easy. The best practice for finding the confidence interval for the 
shape parameter P is to perform a logarithmic transformation on the estimator to improve the 
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symmetry of its sampling distribution, and then approximate the sampling distribution of 
Infy^jby a normal distribution. The eonfidenee bounds then beeome, 


\lowerbound{L), upper bound (U)] = PIW ,f5*W 


(2.4) 


where W = exp / P\ ando-^ = . The value for Vdr{P) is obtained from the 

parameter varianee-covarianee matrix that is calculated in the analysis software using the 
standard maximum likelihood estimates (MLE) [Ref; 3, Appendix B]. For a symmetric 
distribution such as the normal, the adjustment between a two-sided and a one-sided Cl is made 
by replacing with z^ and choosing the appropriate end of the distribution (either the upper 

bound or the lower bound in Eq. 2.4). Similar adjustments are made to compute confidence 

intervals for other parameters and functions of parameters, such as reliability. 

7. One-Sided Confidence Bounds for the Weibull Distribution Using the 
Likelihood Ratio Method 

Confidence intervals based on a symmetric sampling distribution produce equal values in 
the tails of the probability density functions (pdf). However, often the sampling distribution for 

an estimator (such as ) is not symmetric. The usual sampling distribution in most practical 
applications takes a shape similar to the southeast comer example in Figure 3. To find the 
approximate two-sided confidence interval, the relative likelihood is used. The justification for 
this approach is presented in various texts [Ref; 3, p. 627] and leads to the following equation for 
determining a two-sided confidence interval; 

R(0)>txv[-xl.„l2] (2.5) 


where R{6) is the ratio of the likelihood function of a single parameter (theta) divided by the 
likelihood evaluated at the maximum likelihood value of the parameter. Values of the Chi- 
Squared distribution are available in tables in most statistic texts. 

The technique for finding the two-sided interval can be envisioned as plotting the relative 
likelihood (on the left y-axis) against the values of the estimated parameter while simultaneously 
plotting the confidence levels (on the right y-axis). The value of Eq. 2.5 determines a horizontal 
line that intersects the graph of the relative likelihood function (see Figure 5). The vertical lines 
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drawn from the intersections of the horizontal line and the relative likelihood, down to the x-axis, 
determine the confidence interval endpoints. 



Figure 5. 2-Parameter Weibull likelihood distribution showing a 95% confidence level 
(horizontal line) and confidence bounds (vertical lines) of (1.897, 4.772) [Ref; 3, p. 

181] 

(Note that the area is not symmetrically distributed between 
the upper and lower bounds.) 

This thesis is interested in finding the one-sided confidence bound of for reliability based 
on a Weibull distribution as is often used in reliability measurement and warranty calculations. 
A one-sided confidence interval for 0 can be obtained from Eq. 2.5 by replacing a with 2a and 
drawing the horizontal line in Figure 5 at the value where 

R(9)>txv[-xU.,,l2] (2.6) 

Then, choosing the correct endpoint from the x-axis of relative likelihood graph, you can 
obtain either the lower or the upper, one-sided confidence bound. This is the usual practice for 
one-sided likelihood ratio confidence bounds. [Ref; 3, p. 186] 
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This approach extends to distributions that have more than one parameter and to 
functions of parameters, and is discussed extensively by Meeker and Escobar [Ref: 3, Chapter 8]. 
The key point is that the relative likelihood is asymmetric, and so two-sided intervals using it 
have better coverage properties than the FM method. For one-sided confidence intervals, use of 
Equation 2.6 still presents problems as it assumes that the confidence is distributed evenly 
between the upper and lower tails of the region. This is not the case, and is the source of the 
problems for this method when applied to one-sided intervals. 
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III. METHODOLOGY 


A. SIMULATION 

This thesis uses a reliability software paekage called S-Plus [Ref; 7] to simulate small 
sample size experiments with few failures. S-Plus uses an attached life data analysis routine 
called SPLIDA [Ref; 2] that performs the life data analysis. The S-Plus software package was 
chosen because of its wide application in the study of probability and statistics, its built-in life 
data analysis package, and the fact that S-Plus has the capability of user defined ‘functions’ that 
can be run for a large number of trials, hence the basis for the simulation study. 

A simulation code (Appendix A) was written in S-Plus for the Weibull distribution that 
takes the location-scale parameters, beta (JJ) and eta (?/), the number of items under test (n), an 
alpha (a) where (1- a)* 100% is the reliability required for a product (target or contractual), and 
the number of failures (f) recorded during the test time. The code first derives the true value of 
the quantile for the Weibull distribution using the given parameters and uses that value for 
comparing whether the result of each iteration yields a Cl that contains that true value. If the 
true value is contained within the interval, a counter is incremented by one; if it is not contained 
within the interval it is not incremented. The ratio of the count to the number of iterations is an 
unbiased estimate of the probability coverage [Ref; 3, p. 49] and is plotted for each reliability 
value. An example of probability coverage is shown in the figures below. 

Figure 6 is a graph of 40 simulations of a sampling from a normal distribution [Ref; 9] 
with a two-sided 95% Cl for the mean calculated for each simulation. Compare that to Figure 
7, which shows the same simulation run with a 90% Cl. The width of each interval with 90% 
coverage is smaller than that of the 95% Cl. Finally as a comparison. Figure 8 shows a 95% Cl 
but with a larger sample size; it shows a small interval as well. The probability coverage 
represents a useful tool to determine if the value of interest falls within the Cl and to quantify the 
coverage. 
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Figure 6. A 95% confidence interval for normally distributed data with a sample size of 10, 
showing one of the 40 CFs doesn’t contain the true value for an observed coverage of 

95%. [Ref: 9] 



Figure 7. A 90% confidence interval for normally distributed data with a sample size of 10, 
showing four of the 40 intervals do not contain the true value for an observed 

coverage of 90%. [Ref: 9] 
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Figure 8. A 95% confidence interval for normal distributed data with a sample size of 20, 
showing that two of the 40 intervals do not contain the true value for an observed 

coverage of 95%. [Ref; 9] 

The initial S-Plus code determined the probability coverage using both the Fisher-Matrix 
(FM) and the Likelihood Ratio Bound (LRB) methods and compared the two methods by 
simultaneously plotting them on the same graph. Even though both procedures are known to be 
inaccurate for small sample sizes with censored data, Meeker states [Ref: 3, p.l65]. 

The computationally demanding likelihood procedure can be expected to provide 
better intervals (i.e. an actual coverage probability closer to nominal confidence 
level). 

B. APPLICATION 

The purpose of the simulation study was to quantify the bias between the two confidence 
methods and to develop tables showing the error. Then, the method with the greatest error was 
chosen to produce a catalog of graphs representing typical reliabilities often found in military 
contracts. The reliability is a required input to the simulation code that produces a separate 
graph for each value. The reliabilities chosen for this thesis were 99, 98, 97, 96, 95 and 90%. 
The numbers of items available for test (sample size of 10 and 20) establish the test population 
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and the numbers of failures that oecur within each of the test populations (3, 5, 7, and 9) 
establish the amount of censoring. The confidence levels are not a required input to the 
simulations but are produced as a result of the S-Plus code for the two standard methods listed. 

The output of the simulation produces a graph for each of the input variables (six 
reliabilities (times) two populations (times) four levels of censoring) resulting in a catalog of 48 
graphs representing typical values found in most reliability and warranty specifications. Some 
graphs in this thesis display both of the two confidence methods simultaneously on each graph 
(see Figure 9 below for typical graph display) and allows a comparison between the two methods 
and is used to graphically display the error and differences between the two methods. 


Actual vs. Requested Coverage for N= 1200 , n= 10 , f= 3 



Figure 9. Example graph generated in S-Plus using a 1200 iteration Monte Carlo simulation 
showing the simultaneous plots of the FM and LRB methods. 

The dashed line (red) on each graph is the simulation output of the one-sided confidence 
bound using the FM method and the blue line (solid) is the output using the LRB method. Most 
of the available software packages for life data analysis use the normal-approximation (Fisher- 
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Matrix) method with appropriate transformations to achieve symmetry but Meeker and Escobar 
assert that the likelihood ratio method is more accurate [Ref; 3, p. 165]. This thesis supports 
their assertion. 


Actual vs. Requested Coverage for N= 1200 , n= 10 , f= 3 



Figure 10. Expanded view of a typical graph showing the intersection of the actual coverage 

with each method and the resulting requested coverage. 

The catalog of graphs is contained in Appendix B and only shows the more inaccurate 
Fisher-Matrix (EM) method. It is divided into six sections; one section for each reliability value 
with each section sub-divided by the sample size with a graph for each censoring level. The user 
need only turn to the appendix and select the correct reliability, find the sample size of interest, 
and locate the graph with the appropriate number of failures (censoring). The user then draws a 
horizontal line from the y-axis (Actual Coverage) to either one of the two confidence methods 
(solid or dashed lines), then down to the x-axis (Requested Coverage) to read the amount of 
confidence they require to have in order to get what he requests (see Figure 10). The black 
diagonal line across the middle of each graph is the theoretical value where the “Requested 
Confidence” equals the “Actual Confidence”. In other words, if we had a large enough samples 
and no censored data, both of the confidence methods would lie close to that line. 
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Figure 10 shows that as a result of running the simulation, that if the user needed an 
actual 80% one-sided confidence bound, they would have to request a 92% confidence bound 
using the LRB method and would need to request a 96% confidence bound using the FM method 
for this sample size and censoring scheme. 

Figure 11 incorporates a modified S-Plus code that draws the line on the graph and 
displays the adjusted confidence bounds below the x-axis. The graph indicates that for an actual 
90% coverage, you must request 92.16% for the LRB method and 96.4% for the FM method. 


Actual vs. Requested MLE coverage for N= 1200 , n= 52 , and f = 23 



Figure 11. Example graph generated in S-Plus using a modified code that displays the 

required values and automatically draw the lines. 
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IV. RESULTS 


A. INTRODUCTION 

A summary of the relevant features that support the initial propositions of this thesis is 
included in this chapter, along with a detailed explanation of the simulation output using one of 
the figures as an example. In addition, a brief discussion of the S-Plus code written for this 
thesis is presented for users who may need to modify the simulation to include different test 
sample sizes, different distributions, or different amounts of censoring that were not performed 
as part of this study. An example is included at the end of this chapter demonstrating the 
usefulness of the tables and showing the impact of the bias on the acquisition process. 

Several of the graphs and tables included in this section show that as the number of 
censored observations increases (fewer failures), the bias increases. That increase is quantified 
in Tables 1 and 2 for each of the two methods using a sample size of 10, an 80% confidence 
level, and varying the number of failures. Based on the results of the simulation study as shown 
in Tables 1 and 2, the graphical output of the simulation study (Appendix B) was run using the 
Fisher-Matrix method only, as it consistently produced the greatest amount of bias and is usually 
the default method used by the listed software packages. 

Appendix B is divided into six sections; one section for each reliability value. Each 
section is sub-divided by the population size (10 and 20). A graph showing the output of the 
simulation is displayed for each number of failures that occurred during the test period. For 
example, in Appendix B, Sections A and B contain the results of the simulation study for a 
contractual 99% reliability. Section A lists the results for a sample size of ten (10) and shows 
four graphs, one for each of the simulated failures (3, 5, 7, 9), while Section B displays similar 
graphs, but shows the results of using a sample size of twenty (20). 

B. SUMMARY OF GRAPHS 

Several observations are evident and worthwhile mentioning in this section. First, by 
comparing the simulation output of either coverage method in any of the reliability series shown 
in the body of this thesis, one can see the bias in the confidence interval increases (the simulation 
value gets farther away from the diagonal line) as the amount of censoring increases (i.e. number 
of failures in the test sample size decrease from nine to seven to five, etc.). This effect is 
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graphically displayed in Figure 11 below, where all four of the different failures are plotted 
simultaneously on one graph for the Fisher-Matrix (FM) method, using a 98% reliability and a 
sample size of 10. The most error in the confidence bounds occurred when the data is heavily 
censored (i.e., only three failures and seven still working). 


Actual vs. Requested Coverage for N= 1200 , n= 10 , f=3,5,7,9 



Figure 12. A graph showing an increasing error in the confidence bounds as the numbers of 
failures decrease for the Fisher-Matrix Method for 98% reliability and a test 

population of 10. 


Note; The data is the result of 1200 Monte Carlo 
simulations using S-Plus with SPLIDA. The colors 
are in order of closest to the diagonal line to farthest 
away, (green = 9 failures, blue = 7 failures, red = 5 
failures, blaek = 3 failures) 

The second noticeable finding in this simulation study is that the Fisher-Matrix method of 
determining confidence bounds has more bias than the Likelihood Ratio Bounds method (LRB). 
The small sample size coupled with censored data has more of an effect on the FM method than 
the LRB method and supports the theorists who have asserted this point for years. Tables 1 and 
2 quantify the bias for each of the two methods using a sample size of 10, an 80% eonfidenee 
level, and varying the number of failures. 
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Based on the results of the simulation study as shown in Tables 1 and 2, the graphical 
output (Appendix B) of the simulation study only was run using the FM method as it consistently 
produced the greatest amount of bias. The following tables are based on 10000 iterations from 
using the Monte Carlo code written for this thesis. 


Number 

of 

Failures 

Reliability 

99% 98% 97% 96% 95% 90% 

3 

20% 

18.8% 

18.8% 

16.3% 

16.3% 

5% 

5 

17.5% 

16.3% 

13.8% 

13.8% 

13.8% 

11.3% 

7 

15% 

13.8% 

12.5% 

13.8% 

12.5% 

10% 

9 

13.8% 

12.5% 

11.3% 

13.8% 

13.8% 

11.3% 


Table 1. Percent error in the confidence bounds using the Fisher-Matrix Method (FM), a 
sample size of 10, and an 80% (actual) confidence level. (Where the Error = abs 
[(Required-Actual)/Actual] xl00%). 


21 
















Number 

of 

Failures 

Reliability 

99% 98% 97% 96% 95% 90% 

3 

13.8% 

12.5% 

13.8% 

8.8% 

8.8% 

0% 

5 

11.3% 

10% 

11.3% 

8.8% 

11.3% 

6.3% 

7 

8.8% 

11.3% 

11.3% 

7.5% 

6.3% 

6.3% 

9 

6.3% 

8.8% 

19% 

11.3% 

10% 

11.3% 


Table 2. Percent error in the confidence bounds using the Likelihood Ratio Bounds (LRB) 
method, a sample size of 10, and an 80% (actual) confidence level. (Where the Error 
= abs [(Required-Actual)/Actual] xl00%). 

C. ERROR CORRECTION USING THE CATALOG 

The following tables summarize the required value for the FM method that one must 
specify in order to obtain an 80% confidence bound for each of the simulations in this thesis. 
These values can be read directly from the graphs in Appendix B or can be produced by the 
simulation code running the subroutine get.hartley.lcb’ and are representative of the information 
available in the Appendix. The 80% confidence bound was chosen as a demonstration of the 
information available from the catalog. 

Users of this catalog must specify the “Requested” value that is read from the appropriate 
graph that meets their particular needs and meets the small sample criteria with censored data. 
So for a 97% reliability and an 80% confidence for a sample size of 10 with 5 failures, one 
would have to specify an 89% confidence level in their software in order to meet a true 80% 
confidence as required by the specification. 
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Number of 

Reliability 

Failures 

99% 

98% 

97% 

96% 

95% 

90% 

3 

99.99 

99.98 

99.98 

99.97 

99.97 

99.7 

5 

99.04 

98.69 

98.68 

98.17 

98.16 

97.54 

7 

96.84 

96.40 

95.94 

95.71 

96.06 

94.60 

9 

94.20 

93.65 

94.14 

93.76 

93.81 

93.09 


Table 3. Summary of the required confidence using the Fisher-Matrix method (FM) and an 
actual 80% confidence using a Monte Carlo simulation with a sample size of 10 with 
3, 5, 7, & 9 failures for 99%, 98%, 97%, 96%, 95%, and 90% reliability. 


Number of 

Reliability 

Failures 

99% 

98% 

97% 

96% 

95% 

90% 

3 

99.99 

99.98 

99.97 

99.91 

99.80 

95.83 

5 

99.02 

98.63 

98.68 

98.41 

97.91 

95.28 

7 

97.27 

96.48 

96.48 

96.40 

96.18 

94.27 

9 

95.40 

94.85 

94.67 

93.86 

93.85 

92.25 


Table 4. Summary of the required confidence using the Fisher-Matrix method (FM) and an 
actual 80% confidence using a Monte Carlo simulation with a sample size of 20 with 
3, 5, 7, & 9 failures for 99%, 98%, 97%, 96%, 95%, and 90% reliability. 


Although Table 4 above shows values in the 90% reliability column for both three and 
five number of failures, the values are questionable to a degree. The graphs of the simulation for 
these two failures (Appendix B, Section L) show an inflection point in the output data where the 
graph rises above the diagonal line (lower half). This appears to be a result of the heavily 
censored data (only 3 of 20 and 4 of 20 failures) coupled with a low reliability (90%). The 
numbers appear to follow the trends in Table 4 and the upper half of the graphs in Appendix B, 
Section L are of similar magnitude as other graphs in that class, therefore I left the entries in 
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Table 4 and suggest they are reasonable for this thesis. Clearly, however, small numbers of 
failures result in inaccurate coverage probabilities for low reliabilities. This behavior has been 
confirmed by repeated trials and warrants further study. 

D. CODE MODIFICATION FOR SPECIFIC APPLICATIONS 

The S-Plus code that was used in this thesis can be modified to change many of the 
parameters used to describe the life data analysis. In particular, the sample size and number of 
failures can be specifically tailored to accommodate many users. 

The first line of the code is the ‘function’ line and contains all of the relevant parameters 
including number of iterations (N), sample size (n), and number of failures (f), and confidence 
level (g). The function line appears below as used in this thesis. 

get.hartley.lcb <- function(N = 10000, g = 0.95, n = 13, f = 10, alpha = 0.1, plot.it = F) 

E. TEST CASE 

To demonstrate an application of the usefulness of the tables in this thesis and as another 
example of modifying the S-Plus simulation code, this section will replicate the results of a 
worked example in Lawless’ text on lifetime data [Ref; 10, p. 155]. In his text, he worked an 
example using failure times from a small sample of aircraft components with censored data. His 
solution was based on the Weibull distribution and used tables that appeared in a technical report 
[Ref; 11] from Wright-Patterson AFB, OH to obtain a value for 90% reliability and a 95% lower, 
one-sided confidence bound. The exact value of B10C95 obtained using the tables in Reference 
11, was (.105). 

The data shown in the table below are the 10 failure times of aircraft components and the 
three censored times all stopped after the tenth failure. The values of N = 13 and f = 10 were 
used as inputs to the S-Plus simulation code used in this thesis. The function line of the code is 
shown below; 
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function(N = 10000, n = 13, f = 10, alpha = 0.10, plot.it = T) 


Failure (F)/Censored(S) 

Time (hours) 

F 

0.22 

F 

0.5 

F 

0.88 

F 

1 

F 

1.24 

F 

1.33 

F 

1.54 

F 

1.76 

F 

2.5 

F 

3 

S 

3 

S 

3 

s 

3 


Table 5. Aircraft component failure times in hours. [Ref: 10, p. 155] 


The output of the simulation code for the FM method is shown in Figure 12. Using the 
same graphical technique that was described in Figure 9, the ‘required confidence’ was located 
by drawing a horizontal line from the y-axis at the confidence value your specification calls for 
(in this case 95%), and intersecting with the simulation output. Then drawing a vertical line 
down to the x-axis and estimating the new confidence value that you need to request from the 
contractor or testing organization (approximately 99.5%). 
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Actual vs. Requested coverage for N= 10000 , n= 13 , and f = 10 



Figure 13. Output of the S-Plus simulation code using N =13 and f=10 derived from 

Example 4.1.1. [Ref; 10] 

An output of the S-Plus simulation code subroutine ‘get.hartley.lcb’ is shown below. The 
first line matches the exact value of B10C95 as calculated by Lawless [Ref; 10, p. 155] and 
indicates the simulation code is working correctly. The following lines are the final lines of the 
S-Plus output and specifies the ’required’ confidence directly without using the graphs. 

The true 0.1 point is 0.1053 

We have a sample of size 13 and 10 failures. 

To get a 95 percent actual confidence level, you must request a 99.85 percent FM 
confidence level. 

The results of this simulation output are close to the estimated value of the required 
confidence using the visual aid of drawing the line across and down from Figure 13 as shown 
above (approximately 99.5%). The simulation output may provide a more accurate confidence 
value for those who have S-Plus and SPLIDA as analysis tools but the graph technique using the 
graphs in Appendix B is also quite good. 
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As a final verification of the simulation code and another indication that the ‘required’ 
confidence from the simulation is credible, I analyzed the Lawless data in another life-data 
analysis program called Weibull++6 [Ref. 12], In particular, I was interested in obtaining the 
B10C95 ( shorthand for the 95% lower confidence bound for the tenth percentile) value using 
both the original confidence value (95%) , and then recalculating using the value obtained from 
the S-Plus output (either from the graphs or the direct readout from the subroutine 
‘get.hartley.lcb’). 

Using the data from Table 5, the Weibull++6 program provided the following graph 
shown in Figure 14, as well as Tables 6 and 7. 


ReliaSoft's Weibull++ 6.0 - www.Weibull.com 
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Figure 14. The output of the Weibull++6 software package used to determine the B10C95 

value for the S-Plus simulation code. 
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User Input; 


BX% information at; = 

10 

Confidence Bounds Used; 

1-Sided 

Confidence Bounds Method; 

Fisher Matrix 

Confidence Level; = 

0.95 

Weibull++ Output; 


Time; = 

0.459 

Lower Limit; = 

0.205 

Confidence; = 

IS @0.95 


Table 6. Output from the Weibull++6 Life Data Analysis Software paekage used for 

comparison with Example 4.1.1. [Ref; 10, p. 155] 


Table 6 shows that the lower limit using the Fisher-Matrix method was (.205) compared 
to the exact value of (.105) obtained by Lawless [Ref; 10, p. 155]. The B10C95 value using the 
original confidence value of 95% is almost twice the exact value calculated by Lawless! 

The ‘required’ confidence value of 99.85% from the subroutine ‘get.hartley.lcb’ was then 
used in the Weibull++6 software package to determine a new lower limit. Table 7 displays the 
result of the calculations and shows the lower limit of (.1073) using the estimated value from the 
simulation code for the confidence value. The B10C95 now compares favorably to the exact 
value of (.105) as determined by Lawless. 


User Input; 


BX% information at; = 

10 

Confidence Bounds Used; 

1-Sided 

Confidence Bounds Method; 

Fisher Matrix 

Confidence Level; = 

0.9985 



Weibull++ Output; 


Time; = 

0.459 

Lower Limit; = 

0.1073 

Confidence; = 

IS @0.9985 


Table 7. Output from the Weibull++6 Life Data Analysis Software package using the 

confidence value from the S-Plus simulation code for Example 4.1.1. [Ref; 10, p. 155] 


28 




V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

The objective of this thesis was to graphically show that analyzing small sample sizes, 
commonly found in most Developmental Test (DT) scenarios, with software designed for large 
samples, impacts many areas in the acquisition analysis and the small error that has been 
acknowledged for years has a large impact in today’s economy. The results of the simulation 
study in this thesis quantify the bias in one-sided reliability confidence bounds when using small 
samples and censored data. 

Tables 1 and 2 in the previous section are tabular displays of the error in the confidence 
bounds and show that for this simulation study the error can be as much as 20%. The results 
section conclusively shows that the Likelihood Ratio Bounds method (13.8% error) has less bias 
than the more commonly used Fisher-Matrix method (20% error) especially for high reliability 
when the number of failures is small. 

The test case shown in the previous section also illustrates the impact of small sample 
size analysis using actual aircraft component data. The comparison of the B10C95 using the 
original confidence value in the Weibull++6 software package, to the exact value calculated by 
Lawless shows almost a 100% increase (.105 versus .205). This increase can directly translate 
into overestimating inspection times for mission critical components and purchasing and storing 
of unnecessary spares, increasing the logistic tail for combat deployments. 

The bias can be reduced when using standard software packages to analyze small sample 
with censored data by either the graphical technique using the graphs in Appendix B, or running 
the simulation code using S-Plus with the embedded SPLIDA life data routine. The test example 
showed that using either technique can reduce the error from 95.23% (using the 95% confidence) 
to less than 1.9% (using the ‘required’ 99.85% confidence obtained from Figure 13 and the 
output of ‘get.hartley.lcb’). 

Although increasing the number of items under test can increase the confidence, this 
thesis shows that the major influence is the small number of failures that occur during the test 
time. Many of the governments’ contracts specify quantities such as reliability and confidence 
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but we can’t mandate the number of test items because of the constraint to minimize costs and 
the ever-present desire to shorten the test time. An unknown author writes what is known as The 
Paradox of Reliability Analysis , 

The more reliable a product is, the harder it is to get the failure data needed to 

“prove” it is reliable! 

However, we can now argue for better analysis by understanding the bias associated with 
small samples with censored data and modifying our requirements to account for the error. 

B. RECOMMENDATIONS 

Two improvements to the simulation code appeared during the writing of this thesis. The 
first and most important improvement would be to modify the simulation code for other life data 
analysis software packages. Although the use of S-Plus is widespread in industry, many other 
analysis packages exist and are used worldwide. The second improvement would be to write a 
simulation code for other high-speed computer software programs that are not strictly life data 
analysis tools. For example. Dean M. Ford from the Delphi Automotive Systems has developed 
a simulation code for the Weibull distribution that runs on Excel and is much faster than the S- 
Plus code used in this thesis. 

Finally, in addition to improving the speed of the code used in this thesis, this code could 
easily be extended to include other distributions such as exponential, lognormal and gamma. An 
efficient simulation tool or catalog listing all of the location-scale distributions showing the 
required confidence for small samples would be an invaluable tool for anyone dealing with life 
data analysis, reliability, and warranty contracts. 
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APPENDIX A 


This section contains the S-Plus code that was written for this thesis. It performs a Monte 
Carlo simulation for the Weibull function using the number of test items and the number of 
failed units. This code can be copied and used in the function command in S-Plus. After naming 
the function, run it by calling the function and specifying the number of iterations in parentheses 
similar to the following line; > get.hartley.lcb(N = 10000). If you require a specific confidence 
level and would like the S-Plus program to calculate determine that value for you, modify the 
‘get.hartley.lcb code in the function line below by specifying the reliability (alpha), and save 
‘get.hartley.lcb. Now run the function ‘get.hartley.lcb and set the variable ‘g’ to the confidence 
you want. The program will calculate the required confidence level and run the function 
resulting in a plot of the actual versus required confidence for all confidence values. (Note; The 
SPLIDA subroutine must be loaded into the S-Plus code in order for this code to work.) 


####################################### 

# This program is the final working version of Mike Hartley's 

# routine to get equivalent confidence levels for BXLCBYY 

# 

# to use, load the first function. A typical function call is the 

# last line of the code below. 

# 

# 2/29/04; DHO 

# 

# “Team players use S Plus, but it takes the JV longer....” 

# 

######################################### 


get.hartley.lcb<-function(N= 10000, g=.95, n=13, f=10, alpha = 0.1, plot.it=F) 

{ 

#bad parameter traps 
if(alpha >0.5) 

stop(“Try another alpha. \n \n”) 
if(g>=l II g<=0) stop (“Try another g. \n \n”) 
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true.pt<-exp(qsev(alpha)) # the true percentile 
cat(“The true”, alpha, “point is”, true.pt, “\n”) 

test.out<-SingleDistSim(number.sim=N,”WeibuU”,sample.size=n, 
censor.type = “Type 2”, fail.number = f, kprint = 0) #gets a sample from the weibull of 
size N 

mu<-test.out$theta.hat[, 1 ] 
sigma<-test.out$theta.hat[,2] 
var.mu<-test.out$vcvobs[,l] 
var. sigma<-test. out$vcvobs[,3 ] 
covar<-test.out$vcvobs[,2] 

tp<-exp(mu + qsev(alpha)*sigma) # tp is the vector of point estimates from the 
samples 

se.tp<- tp *sqrt(var.mu + 2* qsev(alpha) *covar +(qsev(alpha))^2 * var.sigma) #the 
vector of SEs for tp 

ecl<-(pnorm(tp*log(tp/true.pt)/se.tp)) #the CL necessary to cover the true value for each 
sample 

ecl<-sort(ecl) #the sorted ECLs 
if (plot.it==T){ 

plot(ecl,((l;N)-.5)/N, type='n', xlab='Requested coverage', ylim=c(0,l), ylab='Actual 
coverage', main = paste(“Actual vs. Requested coverage for N=“, N, “, n=“,n,”, and f-‘, f)) 
hnes(ecl,((l;N)-.5)/N, col=2) 
mtext(paste(“Quantiles for p = “,alpha)) 
abline(0,l,col=l) 

legend(0.2,0.9,c(“EM”), lty=c(l,l),col=c(2,2)) 

} 


cat(paste(“We have a sample of size”, n, “ and “, f,” failures. \n”)) 

cat(paste(“To get a”, 100*g, “ percent Hartley confidence level, you must request a”, 
100*quantile(ecl,g), “percent EM confidence level. \n”)) 


} 

get.hartley.lcb(N= 10000, n=13, f=10, alpha = .1, g = .95, plot.it = T) 
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Actual coverage Actual coverage 


APPENDIX B 


RELIABILITY = 99%, SAMPLE SIZE = 10 


Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 5 



Requested coverage 

















Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 9 
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B. 


RELIABILITY = 99%, SAMPLE SIZE = 20 


Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 9 
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C. RELIABILITY = 98 %, SAMPLE SIZE = 10 


Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 9 
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D. 


RELIABILITY = 98%, SAMPLE SIZE = 20 


Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 9 
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E. 


RELIABILITY = 97%, SAMPLE SIZE = 10 


Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 9 
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F. 


RELIABILITY = 97%, SAMPLE SIZE = 20 


Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 9 
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G. 


RELIABILITY = 96%, SAMPLE SIZE = 10 


Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 9 
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H. 


RELIABILITY = 96%, SAMPLE SIZE = 20 


Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 9 
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I. 


RELIABILITY = 95%, SAMPLE SIZE = 10 


Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 9 
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J. 


RELIABILITY = 95%, SAMPLE SIZE = 20 


Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 9 
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K. 


RELIABILITY = 90%, SAMPLE SIZE = 10 


Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 10 , and f = 9 



54 















L. 


RELIABILITY = 90%, SAMPLE SIZE = 20 


Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 3 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 5 
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Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 7 



Actual vs. Requested coverage for N = 10000 , n = 20 , and f = 9 
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